Window-shaping virtual reality system

ABSTRACT

A virtual reality system, comprising an electronic 2d interface having a depth sensor, the depth sensor allowing a user to provide input to the system to instruct the system to create a virtual 3D object in a real-world environment. The virtual 3D object is created with reference to at least one external physical object in the real-world environment, with the external physical object concurrently displayed with the virtual 3D object by the interface. The virtual 3D object is based on physical artifacts of the external physical object.

The present patent application is a continuation of U.S. Patent No.10,643,397, issued May 5, 2020, which is related to and claims thepriority benefit of U.S. Provisional Patent Application No. 62/473,465,filed Mar. 19, 2017 the contents of which are incorporated in theirentirety herein by reference.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent application is a continuation of U.S. patentapplication Ser. No. 15/925,758 filed Mar. 19, 2018, which is related toand claims the priority benefit of U.S. Provisional Patent ApplicationSer. No. 62/473,465 filed Mar. 19, 2017 the contents of which areincorporated in their entirety herein by reference.

STATEMENT REGARDING GOVERNMENT FUNDING

This invention was made with government support under Contract No.IIP-1632154 awarded by the National Science Foundation. The governmenthas certain rights in the invention.

TECHNICAL FIELD

This disclosure generally relates to virtual reality systems, and morespecifically, to virtual reality computer system which allows creationand visualization of shapes directly on a surface of an existingreal-world object in the virtual space.

BACKGROUND

This section introduces aspects that may help facilitate a betterunderstanding of the disclosure. Accordingly, these statements are to beread in this light and are not to be understood as admissions about whatis or is not prior art.

Human capabilities enabled by visual, aural, tactile, kinesthetic, andspatial perceptions are underutilized by digital media for artifactcreation. To cooperate with an increasingly growing world of digitalmanufacturing/fabricating, humans have been forced to use multi-step,tedious, and cumbersome interactions with the associated virtual designand display systems. In such processes, natural modes of thinking andcommunication are fractured because of: (1) switching between tangibleinstruments such as measurement and input devices like keyboards andmice, and (2) limitations in visual perception of virtualrepresentations of physical artifacts. These discontinuities ininteractions are a result of background representations and modalitiesfor information exchange used by these machines and devices. Forexample, the design environment is typically implemented through a 2DWindows-Icons-Menus-Pointers (WIMP) manner which requires extensivetraining and inhibit users ability to create virtual 3D shapes in asimple manner. Moreover, such virtual design environment is isolatedfrom the design context which results in the visual perception ofvirtual contents separates from the physical artifacts and thesurrounding environment. The physical environment often serves as ameans for inspiring, contextualizing, and guiding the designer's thoughtprocess for expressing creative ideas. In early design processes,objects are frequently used as references to explore the space of noveldesigns.

Emerging mobile devices further led to a distributive change in the wayusers create, manipulate, and share digital information. Methods forusing mobile devices for design environments and interfaces have beenrecently proposed. Although, these works leverage the ubiquity of mobiledevices, they haven't addressed the fragmented virtual designenvironment and the physical design context. Recent works have shownthat through-the-screen Augmented Reality (AR) and Mixed Reality (MR)can play a vital role in bridging the gap between the physical anddigital worlds for creative expression of ideas. However, most of theseapproaches use the physical environment mainly as a dormant container ofdigital artifacts rather than as a source of inspiration forfacilitating quick digital prototyping for design ideation. Therefore,improvements are needed in the field.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1a shows a virtual representation wherein users simply draw a curveon the screen. FIG. 1b shows a mapping to a 3D planar curve using apoint cloud. FIG. 1c shows the 3D curve is inflated into a 3D model.FIG. 1d shows how users manipulate the shapes through a multi-touchinteraction scheme. FIG. 1e shows a screenshot illustrating howWindow-Shaping enables quick creation of virtual artifacts foraugmenting the physical environment by borrowing dimensional andtextural attributes from objects.

FIG. 2a shows a virtual display rendering of a circular geometricprimitive and associated inflation function in Window-Shaping. FIG. 2bshows a virtual display rendering of a conical geometric primitive andassociated inflation function in Window-Shaping. FIG. 2c shows a virtualdisplay rendering of a tapered geometric primitive and associatedinflation function in Window-Shaping. FIG. 2d shows a virtual displayrendering of a linear geometric primitive and associated inflationfunction in Window-Shaping.

FIG. 3a shows use of a virtual reality system wherein a user sketches aboundary curve on a physical object in the curve mode. FIG. 3b shows useof the virtual reality system wherein the user sketches a hole curve onthe physical object in the curve mode. FIG. 3c shows use of the virtualreality system wherein the user edits the sketched curves to add localdetails. FIG. 3d shows use of the virtual reality system wherein theuser obtains an inflated circular shape according to one embodiment.

FIG. 4a shows use of a virtual reality system wherein a user creates atapered inflated shape with a template in the curve mode and inflatesit. FIG. 4b shows the use of the virtual reality system wherein the userpatterns the shape. FIG. 4c shows the user exploring complex features.FIG. 4d shows the user further exploring the complex shape according toone embodiment.

FIG. 5a shows use of a virtual reality system wherein a user hascaptured outline and texture of a snail shape by drawing ROI around thephysical object. FIG. 5b shows a circular inflated shape using thecaptured outline. FIG. 5c shows an extruded shape using the capturedoutline and textured with the segmented snail shape according to oneembodiment.

FIG. 6a shows an example of a 2D transformation according to oneembodiment. FIG. 6b shows the use of a 3-finger gesture which allows formodifying the inflation of a 3D shape. FIG. 6c shows an example of a 3Dtransformation according to one embodiment. FIG. 6d shows an example ofa further transformation.

FIG. 7 shows an illustration of plane inference: direct un-projection of2D drawing results a discontinuous curve (right), and projection on theinferred plane (right).

FIG. 8a shows a first step for creating a texture by projecting thebounding rectangle of the 3D planar curve. FIG. 8b shows a second stepfor creating the texture by image skewing. FIG. 8c shows a third stepfor creating the texture by rotation correction. FIG. 8d shows a fourthstep for creating the texture by image cropping.

FIG. 9a shows a first example of the expressive capabilities enabled bythe Window-Shaping design workflow according to one embodiment. FIG. 9bshows a second example of the expressive capabilities enabled by theWindow-Shaping design workflow according to one embodiment. FIG. 9cshows a third example of the expressive capabilities enabled by theWindow-Shaping design workflow according to one embodiment.

FIG. 10a shows first step of a furniture design use case wherein avirtual side-table is created by borrowing the texture from a physicaltable. FIG. 10b shows second step of a furniture design use case whereina virtual side-table is created by borrowing the texture from thephysical table. FIG. 10c shows a third step wherein the surroundingobjects are then used to explore the lamp design. FIG. 10d shows afourth step wherein the lamp design is further captured FIG. 10e shows afifth step wherein GrabCut is applied to capture the outline andtexture. FIG. 10f shows a sixth step wherein GrapCut has been applied.FIG. 10g shows a seventh step wherein a decorative object is formed.FIG. 10h shows an eighth step wherein the object is manipulated in thevirtual environment.

FIG. 11a shows a first step of a “creature” design use case wherein theeyes of the creature are created referring to a helmet. FIG. 11b shows asecond step wherein the creature limbs are created referring to a trashcan. FIG. 11c shows a third step wherein the creature body is createdreferring to a piece of white paper. FIG. 11d shows a fourth stepwherein the creature elements are assembled. FIG. 11e shows a fifth stepwherein the scales of the creature are created from a scaled mat. FIG.11f shows a sixth step wherein the scales are patterned on the body ofthe creature.

FIG. 12a shows a first step in a process wherein a virtual shape iscreated on a metal shelf being placed as an arm-rest on the sides of achair. FIG. 12b shows a second step in the process wherein the virtualshape is created on the metal shelf being placed as an arm-rest on thesides of the chair. FIG. 12c shows a third step in the process whereinthe virtual shape is created on the metal shelf being placed as anarm-rest on the sides of a chair. FIG. 12d shows a fourth step in theprocess wherein the virtual shape is created on a metal shelf beingplaced as an arm-rest on the sides of the chair. Placing a box at theappropriate location on the seat allows for proper placement andorientation of the handle in a simple manner.

FIG. 13 shows a virtual reality system according to one embodiment.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of thedisclosure, reference will now be made to the embodiments described inthis description and specific language will be used to describe thesame. It will nevertheless be understood that no limitation of the scopeof the disclosure is thereby intended, such alterations and furthermodifications in the illustrated embodiments, and such furtherapplications of the principles of the disclosure as described thereinbeing contemplated as would normally occur to one skilled in the art towhich the disclosure relates.

In the following description, some aspects will be described in termsthat would ordinarily be implemented as software programs. Those skilledin the art will readily recognize that the equivalent of such softwarecan also be constructed in hardware, firmware, or micro-code. Becausedata-manipulation algorithms and systems are well known, the presentdescription will be directed in particular to algorithms and systemsforming part of, or cooperating more directly with, systems and methodsdescribed herein. Other aspects of such algorithms and systems, andhardware or software for producing and otherwise processing the signalsinvolved therewith, not specifically shown or described herein, areselected from such systems, algorithms, components, and elements knownin the art. Given the systems and methods as described herein, softwarenot specifically shown, suggested, or described herein that is usefulfor implementation of any aspect is conventional and within the ordinaryskill in such arts.

The presently disclosed system 1800 and interface may comprise, forexample, a hand-held device (e.g., Google Tango), that serves as a localinterface between the physical environment and the user. Thesimultaneous localization and mapping (SLAM) algorithm available withthe Tango API allows for the acquisition of a point-cloud of the scenewith respect to the global coordinate system. The resulting RGB-XYZ dataallows users to implicitly define planes on any physical surface bysimply drawing on the physical scene. Any touch input on the devicescreen can be unprojected on the physical environment to obtain a 3Dpoint along with its normal in the global (world) coordinate system (asshown in FIG. 1). This helps users define a plane at any recognizedpoint on a physical surface. Below, we describe the design goals,modeling metaphor, and user interactions.

The presently disclosed system 1800 provides integration of physicalobjects into the design process, and supports quick design ideation byallowing users to (a) quickly create 3D geometry in reference tophysical artifacts, (b) borrow shape and appearance from physicalartifacts to re-purpose them for design exploration, and (c) inspect thevirtual artifacts in the physical context from different views in orderto make design modifications, as discussed further below.

The design workflow behind the presently disclosed system 1800 comprisesthree interaction modes, namely: (a) inspection mode, (b) curve mode,and (c) shape mode. The inspection mode is a default state that allowsfor basic operations such as adding new shapes to the environment,inspecting the shape created by users and looking around the physicalenvironment during the design process. Once the intent for shapeaddition is detected, the curve mode is activated and the user selectsone of the four primitive types (see FIG. 2) offered in the application.Following this, the user can create sketches in three ways: drawing,selecting from template curves, or using GrabCut to extract outlines ofphysical shapes. Additionally, the user can also re-shape, rotate,translate, and scale the curve on the plane using one and two fingerinteractions. Upon finalizing the curve, an inflated shape is createdand the application is set to the stand-by mode with the inflated shaperendered on the scene. While in the inspection mode, selecting anexisting shape activates the shape mode. In the shape mode, the user canperform five operations on the selected mesh, namely, 3D placement,planar transformations, copying, patterning, and deletion. After theshapes being created, the system 1800 allows users to edit shapes fromdifferent perspectives and various distances.

Projective Sketching:

The presently disclosed system 1800 allows for direct one finger drawingon the tablet screen. The user first selects a primitive type andsubsequently draws a sketch on the tablet. Once finalized, the sketchedcurve is un-projected on the physical scene and is converted andrendered in the scene as a 3D inflated mesh (FIG. 3). In one embodiment,all curves reside in a plane that is implicitly estimated from theuser's 2D drawing. The first curve drawn by the user is by default theone and the only boundary curve. Once a boundary curve has been defined,multiple hole curves can be drawn inside the boundary (FIG. 3(a,b)).

Placing Curve Template:

As an alternative to direct drawing, the presently disclosed system 1800also provides a set of curve templates (e.g. polygonal primitives, holepatterns) that users can select from (FIG. 4). Once a template isselected, users can simply place them on any surfaces of physical orvirtual objects using a single-tap gesture. The curve is placed on afitted 3D plane around the single-tapped location. The curve templatefeature allows for quick exploration of complex ideas with minimalinteraction effort (FIG. 4(c,d)).

Capturing Outlines:

The presently disclosed system 1800 also allows users to extract theoutline of the object from the scene in the image space (FIG. 5). Thisfeature makes use of the well-known GrabCut algorithm. for imagefore-ground detection. User first defines a region of interest (ROI)with free finger drawing, then within the ROI, the system 1800 detectsthe contour of the foreground and use it to create the inflated mesh.This automatic approach of shape creation enables users to directly usethe visual representation (outline shape and texture) of a physicalobject and re-purposing it in 3D form in their own designs.

Editing Shapes:

Users can edit a shape by simply editing the input curve (FIG. 6(c)).Window-Shaping uses the over-sketching interaction technique to enableintuitive and quick curve editing. This interaction allows users tomodify a shape's geometry, add details, or improve its appearance. Userscan simply draw a new curve segment in proximity to a desired existingcurve segment using one finger interactions.

The use of RGB-XYZ representation enables multi-scale curve editing,i.e. the capability to vary the distance between an object and thetablet while still maintaining the dimensions of the curve in physicalspace. Moving the tablet closer to a desired region allows for precisevirtual operations in screen-space allowing users to create finefeatures on curves. On the other hand, moving the tablet away from aphysical surface allows for better overview that is valuable for coarseoperations such as placing shapes and curve templates on desiredlocations (FIG. 6 (b,c)).

Inflating and Deflating:

The system 1800 provides 3-finger pinch/spread gesture for allowingusers to inflate or deflate a 3D mesh. The users pinch to inflate whichmimics pulling material out from the screen, and spread to deflate aspushing into the screen. We perform an additive inflation, e.g., theadditive amount is a function of the area of the triangle formed by 3finger tips in pixel space.

Rotations and Scaling:

Two-finger rotate and pinch/spread are used for rotating and scaling theshape respectively. The rotation angle and pinch distance are bothcalculated in pixel space. These gestures can be applied either directlyto the 3D shape (in the shape mode or to the underlying curve of theshape (in the curve mode. The two-finger interaction constrains allrigid transformation the plane of the curve.

Translation:

The in plane translation is performed by simply dragging a shape usingone finger. This allows for precise placement of the shape on the planeof its underlying curve. To maintain the consistency of dimensionalperception, we project the finger movement onto the underlying planeinstead of using constant mapping between pixel space and physicalspace.

Placement:

Shape placement allows users to directly transfer a selected 3D shape toany point in the scene by using a one-finger tap gesture. Here the 3Dshape is both translated to the specified point and re-oriented alongthe normal at this point. Similarly to placing the template curve, theusers can place a new virtual object on the physical scene as well as onan existing virtual object based on the closest point of selection. Thismaintains a perceptual depth consistency during interactions.

Auxiliary Operations In addition to geometric operations, the presentlydisclosed system 1800 provides operations such as copying and deleting ashape. We also implemented a manual patterning operation. Users canselect and make copy/pattern of the shape at arbitrary locations byusing the shape placement gesture.

During the over-sketching operation, the presently disclosed system 1800automatically update the texture image to maintain the visualconsistency. In contrast, the texture is not updated during the rigidtransformation of shape. This allows for re-purposing the appearancefrom one object to another object. Additionally, the system 1800provides an option where the user can explicitly choose to update thetexture even during rigid transformations. This can be a helpful whenusers are experimenting with different backgrounds for the same shape.

Curve Processing:

The system allows all curves to be closed, simple and oriented.Additionally, the system 1800 also prefers the curves to be smooth whilepreserving features such as corners. Following this guideline, thesystem performs the processing in four steps. (1) We apply anexponential smoothing filter to the points drawn by the user. This isimplemented while the user is drawing the curve on the screen. (2) Wedetermine if the curve is open or closed based on a distance thresholdbetween the end-points of the curve input. We discard an open curve asan invalid input. (3) For a closed curve, we perform an equidistantcurve re-sampling (4) All boundary curves are required to be orientedcounter-clockwise (positive bounded area) while all holes must beclockwise (negative). The system 1800 corrects the wrong curveorientation by reversing the order of points.

Plane Inference and Un-Projection:

With the organized RGB-XYZ data, the system 1800 first un-projects eachpoint of the processed curve into the physical space. The system 1800obtains 3D position of each point, fit plane within the neighborhood ofeach point, and get associated normal. Based on the standard deviationof the distances between adjacent points, the system 1800 categorizesthe drawings into (1) continuous curve; and (2) discontinuous curve. For(1) the system 1800 determines the plane by averaging the position andnormals of these points. However for (2), this simple averaging resultsunpredictable planes as shown in FIG. 7. To address this issue, thesystem 1800 clusters the points into groups with small euclideandistances and normal differences, e.g., continuous segments on the sameplane and select the segment with most points. Then, the plane isdetermined by the points on this segment.

Mesh Generation:

Given the processed boundary and hole curves, the inflated meshgeneration is performed in three steps: (1) computing two symmetricallyaligned open (half) meshes bounded by the curves (boundary and holes)through constrained delaunay triangulation (CDT), (2) topologicallystitching these two open meshes to create a closed mesh, and (3)inflating the top half mesh using the distance transform function. CDTis implemented using the poly2tri library Unlike extruded primitive, theround, conical and tapered primitives require internal sampling ofpoints before CDT. For this, the system 1800 instantiates a uniformequilateral point sampling inside the region bounded by the curves.These points are added as Steiner points for obtaining a regularlysamples triangulation.

Texture Computation:

In one embodiment, the system 1800 implements texture generation for theinflated mesh using openCV in four steps (FIG. 8). Given theun-projected coordinates of a 3D planar curve, the system first computesits bounding rectangle in 3D space and project it on the image space.Then, the system applies a skew transformation on this image with theconstraint that the projected bounding rectangle is axis-aligned withthe image. In the third step the system 1800 rotates the skewed image tocorrect the residual angle between the rectangle and the image. Finally,the system 1800 crops the image using this projected bounding rectangleto obtain the texture image of the inflated mesh.

The design work flow and interactions implemented using the presentlydisclosed system 1800, can potentially cater to different kinds ofdesign contexts. As nonlimiting examples, four distinct design patternsare described below for the three examples from FIG. 9.

The most fundamental design capability offered by the presentlydisclosed system 1800 is creating new geometric features on existingobjects. These existing objects can be both physical and virtualobjects. For instance, in an interior design scenario, a user could addcomplementary features to a piece of furniture (FIG. 10(a,b)) and alsocreate virtual additions to the scene by adding new assemblies to thesurrounding area (FIG. 10(g,h)).

Re-purposing Physical Objects: By re-purposing, we mean the use of boththe shape and the appearance of a physical object to create a new designfeature. Here, the GrabCut algorithm serves as a means for designinspiration by allowing users to borrow shape and appearance from in thesurrounding environment can be used to re-purpose both the shape andappearance of a physical object for direct use in an existingmixed-reality scene.

Using Physical Objects as Spatial References In situations where userswish to fill in a blank space to augment a physical product, it can behelpful to use another physical object to define a reference plane (FIG.12). The user of objects as references enables a direct, tangible, andspatially coherent way of designing in context. Once the virtual objectis placed appropriately, the reference object can simply be removed fromthe scene.

Using Physical Objects as Visual References The variety of ideasgenerated during early design depends not just on the geometric aspectsbut also the appearance of the design. The appearance can serve for bothaesthetic as well as functional purposes (such as materialspecification). In Window-Shaping, users can quickly experiment with theappearance of a virtual model. Such experiments can be performed eitherby transferring the virtual shape to a new location and re-texturing orby simply changing the background texture of a sketched curve (FIG. 11).

FIG. 13 is a high-level diagram showing the components of one example ofthe system 1800 for analyzing data and performing other analysesdescribed herein, and related components. The system 1800 includes aprocessor 1886, a peripheral system 1820, a user interface system 1830,and a data storage system 8. The peripheral system 1820, the userinterface system 1830 and the data storage system 1840 arecommunicatively connected to the processor 1886. Processor 1886 can becommunicatively connected to network 1850 (shown in phantom), e.g., theInternet or a leased line, as discussed below. It shall be understoodthat the system 1820 may include multiple processors 1886 and othercomponents shown in FIG. 13. The virtual reality and object datadescribed herein may be obtained using network 1850 (from one or moredata sources), peripheral system 1820 and/or displayed using displayunits (included in user interface system 130) which can each include oneor more of systems 1886, 1820, 1830, 1840, and can each connect to oneor more network(s) 1850. Processor 1886, and other processing devicesdescribed herein, can each include one or more microprocessors,microcontrollers, field-programmable gate arrays (FPGAs),application-specific integrated circuits (ASICs), programmable logicdevices (PLDs), programmable logic arrays (PLAs), programmable arraylogic devices (PALs), or digital signal processors (DSPs).

Processor 1886 can implement processes of various aspects describedherein. Processor 186 can be or include one or more device(s) forautomatically operating on data, e.g., a central processing unit (CPU),microcontroller (MCU), desktop computer, laptop computer, mainframecomputer, personal digital assistant, digital camera, cellular phone,smartphone, or any other device for processing data, managing data, orhandling data, whether implemented with electrical, magnetic, optical,biological components, or otherwise. Processor 186 can includeHarvard-architecture components, modified-Harvard-architecturecomponents, or Von-Neumann-architecture components.

The phrase “communicatively connected” includes any type of connection,wired or wireless, for communicating data between devices or processors.These devices or processors can be located in physical proximity or not.For example, subsystems such as peripheral system 1820, user interfacesystem 1830, and data storage system 1840 are shown separately from thedata processing system 1886 but can be stored completely or partiallywithin the data processing system 1886.

The peripheral system 1820 can include one or more devices configured toprovide digital content records to the processor 1886. For example, theperipheral system 1820 can include digital still cameras, digital videocameras, cellular phones, or other data processors. The processor 1886,upon receipt of digital content records from a device in the peripheralsystem 1820, can store such digital content records in the data storagesystem 1840.

The user interface system 1830 can include a touchscreen, hand-heldstylus, mouse, a keyboard, another computer (connected, e.g., via anetwork or a null-modem cable), or any device or combination of devicesfrom which data is input to the processor 1886. The user interfacesystem 1830 also can include a display device, a processor-accessiblememory, or any device or combination of devices to which data is outputby the processor 1886. The user interface system 1830 and the datastorage system 1840 can share a processor-accessible memory.

In various aspects, processor 1886 includes or is connected tocommunication interface 1815 that is coupled via network link 1816(shown in phantom) to network 1850. For example, communication interface1815 can include an integrated services digital network (ISDN) terminaladapter or a modem to communicate data via a telephone line; a networkinterface to communicate data via a local-area network (LAN), e.g., anEthernet LAN, or wide-area network (WAN); or a radio to communicate datavia a wireless link, e.g., WiFi or GSM. Communication interface 1815sends and receives electrical, electromagnetic or optical signals thatcarry digital or analog data streams representing various types ofinformation across network link 1816 to network 1850. Network link 1816can be connected to network 1850 via a switch, gateway, hub, router, orother networking device.

Processor 1886 can send messages and receive data, including programcode, through network 1850, network link 116 and communication interface1815. For example, a server can store requested code for an applicationprogram (e.g., a JAVA applet) on a tangible non-volatilecomputer-readable storage medium to which it is connected. The servercan retrieve the code from the medium and transmit it through network1850 to communication interface 1815. The received code can be executedby processor 1886 as it is received, or stored in data storage system1840 for later execution.

Data storage system 1840 can include or be communicatively connectedwith one or more processor-accessible memories configured to storeinformation. The memories can be, e.g., within a chassis or as parts ofa distributed system. The phrase “processor-accessible memory” isintended to include any data storage device to or from which processor186 can transfer data (using appropriate components of peripheral system1820), whether volatile or nonvolatile; removable or fixed; electronic,magnetic, optical, chemical, mechanical, or otherwise. Exemplaryprocessor-accessible memories include but are not limited to: registers,floppy disks, hard disks, tapes, bar codes, Compact Discs, DVDs,read-only memories (ROM), erasable programmable read-only memories(EPROM, EEPROM, or Flash), and random-access memories (RAMs). One of theprocessor-accessible memories in the data storage system 1840 can be atangible non-transitory computer-readable storage medium, i.e., anon-transitory device or article of manufacture that participates instoring instructions that can be provided to processor 186 forexecution.

In an example, data storage system 1840 includes code memory 1841, e.g.,a RAM, and disk 1843, e.g., a tangible computer-readable rotationalstorage device such as a hard drive. Computer program instructions areread into code memory 1841 from disk 1843. Processor 186 then executesone or more sequences of the computer program instructions loaded intocode memory 1841, as a result performing process steps described herein.In this way, processor 1886 carries out a computer implemented process.For example, steps of methods described herein, blocks of the flowchartillustrations or block diagrams herein, and combinations of those, canbe implemented by computer program instructions. Code memory 1841 canalso store data, or can store only code.

Various aspects described herein may be embodied as systems or methods.Accordingly, various aspects herein may take the form of an entirelyhardware aspect, an entirely software aspect (including firmware,resident software, micro-code, etc.), or an aspect combining softwareand hardware aspects These aspects can all generally be referred toherein as a “service,” “circuit,” “circuitry,” “module,” or “system.”

Furthermore, various aspects herein may be embodied as computer programproducts including computer readable program code stored on a tangiblenon-transitory computer readable medium. Such a medium can bemanufactured as is conventional for such articles, e.g., by pressing aCD-ROM. The program code includes computer program instructions that canbe loaded into processor 1886 (and possibly also other processors), tocause functions, acts, or operational steps of various aspects herein tobe performed by the processor 1886 (or other processor). Computerprogram code for carrying out operations for various aspects describedherein may be written in any combination of one or more programminglanguage(s), and can be loaded from disk 1843 into code memory 1841 forexecution. The program code may execute, e.g., entirely on processor1886, partly on processor 186 and partly on a remote computer connectedto network 1850, or entirely on the remote computer.

The invention is inclusive of combinations of the aspects describedherein. References to “a particular aspect” and the like refer tofeatures that are present in at least one aspect of the invention.Separate references to “an aspect” (or “embodiment”) or “particularaspects” or the like do not necessarily refer to the same aspect oraspects; however, such aspects are not mutually exclusive, unless soindicated or as are readily apparent to one of skill in the art. The useof singular or plural in referring to “method” or “methods” and the likeis not limiting. The word “or” is used in this disclosure in anon-exclusive sense, unless otherwise explicitly noted.

The invention has been described in detail with particular reference tocertain preferred aspects thereof, but it will be understood thatvariations, combinations, and modifications can be effected by a personof ordinary skill in the art within the spirit and scope of theinvention.

The invention claimed is:
 1. A virtual reality system, comprising: adisplay interface; a camera; and a processor, the processor configuredto accept input to the system to instruct the system to create a virtual3D object in a real-world environment displayed on the displayinterface, wherein the virtual 3D object is created with reference to atleast one external physical object in the real-world environment, saidexternal physical object concurrently displayed with the virtual 3Dobject by the interface, wherein the processor is configured to receiveinput from a user and modify the virtual 3D object to match one or moreattributes of the physical object or another physical object in thereal-world environment.
 2. The system of claim 1, wherein the system isconfigured to interpret multi-touch input from the user via the displayinterface, the multi-touch input interpreted by the processor withspatial context using a relative location of the display interface withthe real-world environment.
 3. The system of claim 1, wherein themodified virtual 3D object is based on constraints or modificationssupplied by the user.
 4. The system of claim 1, wherein the displayinterface comprises an electronic display touchscreen and is activatedby human touch, a stylus contact, or a combination thereof.
 5. Thesystem of claim 1, wherein a shape of the virtual 3D object is createdbased on a user sketch input on the display interface, the sketch inputinterpreted by the system as planar curves with respect to a physicalenvironment context.
 6. The system of claim 5, wherein the planar curvesare defined as a 2D mesh mapped on to a physical object, and the 2D meshis inflated to form the 3D virtual object.
 7. The system of claim 1,wherein the system is configured to allow the user to select a shapefrom a menu of template shapes to create the virtual 3D object.
 8. Thesystem of claim 1, wherein the processor is configured to allow the userto select an object of interest in the physical world environment beingdisplayed on the display interface, wherein the system extracts anoutline and corresponding texture of the object of interest to createthe virtual 3D object.
 9. The system of claim 1, wherein the processoris configured to allow the user to scale, translate, rotate, and/or copythe 3D virtual object.
 10. The system of claim 1, wherein the processoris configured to allow the user to edit curves in the 3D virtual objectwith over-sketching.
 11. The system of claim 1, wherein to modify thevirtual 3d object, the processor is configured to retexture the 3Dvirtual object with a texture copied from a physical world object beingdisplayed on the display interface.